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The  Department  of  Defense  (DoD)  prides  itself  on  pro¬ 
viding  "world  class"  training  and  continuing  education  to 
its  military  employees  and  would  like  to  expand  that  repu¬ 
tation  to  its  civilian  employees.  But  just  what  does  "world 
class"  mean  in  the  civilian  context  and  how  can  the  DoD 
best  measure  the  quality  of  its  efforts?  A  recently  released 
RAND  study.  Ensuring  the  Quality  and  Productivity  of 
Education  and  Professional  Development  Activities:  A  Review  of 
Approaches  and  Lessons  for  DoD,  points  out  that  the  DoD's 
education  and  training  efforts  are  highly  decentralized  and 
suggests  that  defense  policymakers  can  gauge  the  caliber 
of  those  efforts  by  choosing  one  of  four  assessment 
approaches. 

BACKGROUND 

With  some  700,000  civilian  employees,  the  Department 
of  Defense  is  the  single  largest  employer  of  civil  service 
workers  in  the  U.S.  government.  Over  the  years,  many  of 
these  civilian  employees  have  taken  advantage  of  a  myriad 
of  education  and  professional  development  courses  that 
the  DoD  provides.  Today,  the  department  offers  more  than 
100  of  these  courses  to  civilian  employees.  Many  of  these 
are  sponsored  by  the  Office  of  the  Secretary  of  Defense, 
which  directs  some  20  institutions  (such  as  the  Naval 
Postgraduate  School)  and  36  programs  (such  as  the 
Defense  Acquisition  Career  Development  Program).  Many 
more  are  provided  by  other  DoD  offices  and  agencies.  And 
still  more  civilian  education  and  professional  development 
takes  place  within  each  military  service. 

To  get  a  handle  on  these  various  efforts,  the  DoD  in 
1998  established  the  Office  of  the  Chancellor  for  Education 
and  Professional  Development,  chartering  it  to  serve  as  the 


"principal  advocate  for  the  academic  quality  and  cost-  .  , 
effectiveness  of  all  DoD  civilian  education  and  professional 
development  activities."  The  Chancellor's  office  is  respon¬ 
sible  for  ensuring  that  all  such  education  and  professional 
development  activities  achieve  appropriate  standards  of 
quality  and  productivity. 

IDENTIFYING  ASSESSMENT  ALTERNATIVES 

Shortly  after  its  creation,  the  Chancellor's  office  asked 
RAND's  National  Defense  Research  Institute  (NDRI)  to 
help  identify  promising  approaches  DoD  might  use  to 
assess  its  educational  programs.  RAND  approached  this 
task  by  reviewing  the  relevant  literature  and  analyzing  the 
methods  that  corporations,  state  governments,  and  univer¬ 
sities  use  to  assess  education  and  professional  develop¬ 
ment  programs.  In  addition,  it  interviewed  assessment 
experts  and  visited  organizations  that  conduct  such  evalu¬ 
ations. 

RAND  found  that  the  Chancellor's  office  might  struc¬ 
ture  such  evaluations  in  several  ways,  which  are  outlined 
in  the  table  on  the  back  page.  It  can  allow  individual  DoD 
institutions  or  programs  to  review  the  quality  and  produc¬ 
tivity  of  their  efforts,  with  no  outside  involvement 
(Approach  1).  A  related  approach  is  for  such  a  review  pro¬ 
cess  to  be  monitored  by  the  Chancellor's  office  or  a  third 
party  (Approach  2).  Alternatively,  the  Chancellor's  office 
or  a  third  party  can  take  on  the  entire  review  process  itself 
(Approach  3).  Or  the  office  can  focus  not  on  the  institutions 
or  programs  but  on  their  outcomes  and  attempt  to  measure 
and  certify  student  achievement  (Approach  4).  Each 
approach  has  strengths  and  weaknesses;  each  works  well 
in  a  different  setting. 
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Strengths  and  Weaknesses  of  the  Four  Assessment  Approaches 


Approach 

Strengths 

Weaknesses 

APPROACH  1 

Provider  designs  the  assess¬ 
ment  process  and  conducts  the 
assessment 

•  Accommodates  differences  among 
institutions  because  it  is  flexible 

•  Provides  a  stimulus  to  self* 
improvement 

•  Is  less  suitable  for  accountability  purposes 

•  Is  not  useful  for  assessing  system-level  needs 

APPROACH  2 

Provider  conducts  the  assess¬ 
ment;  intermediary  reviews 
provider's  assessment  process 

•  Can  accommodate  diversity  of 
institutions  because  it  is  flexible 

•  Promotes  program  improvement 

•  Is  less  suitable  for  accountability  purposes 

•  Can  serve  to  promote  quality  but  not  to  ensure  it 

APPROACH  3 

intermediary  designs  the 
assessment  process  and  con¬ 
ducts  the  assessment 

•  Provides  an  independent  check  on 
quality 

•  Is  well-suited  for  accountability 

•  Can  focus  on  system-level  goals 

•  May  be  overly  standardized  to  reflect 
differences  among  institutions 

•  May  be  driven  by  goals  that  have  little  relation 
to  the  quality  of  education 

•  May  lead  to  institutional  resistance 

•  May  have  little  effect  on  quality  improvement 

APPROACH  4 

Provider  or  intermediary 
assesses  student  competencies 

•  Focuses  on  measuring  student 
learning 

•  Relates  student  learning  to  work¬ 
place  competencies 

•  Is  a  time-consuming  and  expensive  process 

•  Involves  competencies  that  are  less  defined  and 
more  abstract  and  are  therefore  difficult  to 
measure 

•  May  be  more  suitable  for  professional  education 
and  training  than  traditional  academic  institutions 

THE  ACADEMIC  AUDIT  APPROACH:  AN  INTRIGUING 
ALTERNATIVE 

Given  that  the  DoD  system  of  education  and  profes¬ 
sional  development  is  highly  complex  and  decentralized, 
and  that  the  Chancellor's  office  has  little  formal  authority 
over  the  organizations  providing  courses,  NDRI  found  that 
Approach  2 — involving  an  intermediary — appears  to  be 
the  most  promising.  An  intermediary  is  an  organization 
other  than  the  customer  or  provider  of  the  education  and 
could  be  the  Chancellor's  office,  other  DoD  entities,  or 
non-DoD  organizations.  Such  intermediaries  would  be 
responsible  for  evaluating  the  processes  that  individual 
departments  and  schools  employ  to  assess  the  quality  and 
productivity  of  their  educational  efforts.  This  practice 
would  be  similar  to  quality  improvement  efforts  that  have 
been  used  in  the  business  world  for  the  last  25  years  and 
that  were  adopted  by  the  International  Organization  for 
Standardization  in  the  1980s  to  certify  that  manufacturing 
companies  worldwide  adhere  to  certain  quality  standards. 
This  practice  also  would  be  similar  to  academic  audits  that 


increasingly  are  being  used  in  other  education  settings. 
Such  audits  typically  are  conducted  by  intermediary  orga¬ 
nizations  and  focus  on  assuring  that  providers  of  education 
have  effective  processes  in  place  to  measure  their  own 
quality  and  engage  in  ongoing  self-improvement. 

The  study  also  urged  that  the  DoD  look  beyond  assess¬ 
ments  of  existing  education  and  development  efforts. 
Following  the  lead  of  many  corporations  and  educational 
institutions,  a  clear  link  between  education  and  profession¬ 
al  development  on  the  one  hand  and  the  basic  mission  of 
the  DoD  on  the  other  is  needed.  The  Chancellor's  office 
should  advocate  for  the  development  of  a  central  learning 
organization  within  the  DoD  that  would  be  modeled  after  a 
corporate  learning  organization  or  state  higher  education 
coordinating  board.  Such  a  move  would  be  challenging:  It 
would  require  high-level  DoD  support  and  substantial  col¬ 
laboration  among  a  range  of  stakeholders,  including  other 
organizations  within  the  Department  of  Defense  responsi¬ 
ble  for  workforce  planning  and  personnel  policies. 
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